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Examiner's Detailed Office Action 

1 . This office action is responsive to application 09/853,191, filed May 10, 2001. 

2. Claims 1-24 have been examined. 



3. Examiner acknowledges applicants' submission of prior art and information disclosure. 
Nevertheless, applicant is respectfully remind of the ongoing Duty to disclose 37 C.F.R. 1 .56 
all pertinent information and material pertaining to the patentability of applicant's claimed in- 
vention, by continuing to submitting in a timely manner PTO-1449, Information Disclosure 
Statement (IDS) with the filing of applicant's of application or thereafter. 

St 

Drawings 

4. The formal drawings have been reviewed by the United States Patent & Trademark 
Office of Draftperson's Patent Drawings Review. 
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Specification 

5. The specification has not been checked to the extent necessary to determine the presence 
of all possible minor errors. Applicant's cooperation is required in correcting any errors of which 
applicant may become aware in the specification. 

Claim Objections 

6. Claims 4-13 are objected to as being dependent upon a rejected base claim, but would be 
allowable if rewritten in independent form including all of the limitations of the base claim and 
any intervening claims. 

Claim Interpretation 

7. Office personnel are to give claims their "broadest reasonable interpretation" in light 
of the supporting disclosure. In re Morris, 127 F.3d 1048, 1054-55, 44 USPQ2d 1023, 1027-28 
(Fed. Cir. 1997). Limitations appearing in the specification but not recited in the claim are not 
read into the claim. In re Prater, 415 F.2d 1393, 1404-05, 162 USPQ 541, 550-551(CCPA 
1969). See *also In re Zletz, 893 F.2d 319, 321-22, 13 USPQ2d 1320, 1322(Fed. Cir. 1989) 
("During patent examination the pending claims must be interpreted as broadly as their terms 
reasonably allow. . . . The reason is simply that during patent prosecution when claims can be 
amended, ambiguities should be recognized, scope and breadth of language explored, and clari- 
fication imposed. ... An essential purpose of patent examination is to fashion claims that are 
precise, clear, correct, and unambiguous. Only in this way can uncertainties of claim scope be 
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removed, as much as possible, during the administrative process.")- see MPEP § 2106 

Claim Rejections - 35 USC § 102 

8. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 1 02 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in a patent granted on an application for patent by another filed in the United 
States before the invention thereof by the applicant for patent, or on an international application by another who 
has fulfilled the requirements of paragraphs (1), (2), and (4) of section 371(c) of this title before the invention 
thereof by the applicant for patent. 

9. The changes made to 35 U.S.C, 102(e) by the American Inventors Protection Act of 1999 
(AIPA) and the Intellectual Property and High Technology Technical Amendments Act of 2002 
do not apply when the reference is a U.S. patent resulting directly or indirectly from an 
international application filed before November 29, 2000. Therefore, the prior art date of the 
reference is determined under 35 U.S.C. 102(e) prior to the amendment by the AIPA (pre-AIPA 
35 U.S.C. 102(e)). 

10. Claims 1-3, 14-15, & 22-24 is rejected under 35 U.S.C. 102(e) as being anticipated by 
Li et aL (USPAP Pub No.: US 2002/0161747 Al) Filed: Mar. 13, 2001; 

Pub Date: Oct. 31, 2002. 
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Regarding claim 1: 

Li et al teaches, 

A computer system having one or more memories and one or more central processing units 
(CPUs), the system comprising: 

one or more multimedia items, stored in the memories, each multimedia item having two or more 
disparate modalities, the disparate modalities being at least one or more visual modalities and 
one or more textual modalities; [Abstract] "Text features corresponding to pieces of media 
content (e.g., images, audio, multimedia content, etc) are extracted from media content sources. 
One or more text features (e.g., one or more words) for a piece of media content are extracted 
from text associated with the piece of media content and text feature vectors generated therefrom 
and used during subsequent searching. Additional low-level feature vectors may also be 
extracted from the piece of media content and used during the subsequent searching. Relevance 
feedback can also be received from a user(s) identifying the relevance of pieces of media content 
rendered to the user in response to his or her search request. The relevance feedback is logged 
and can be used in determining how to respond to subsequent search requests, such as by 
modifying feature vectors (e.g., text feature vectors) corresponding to the pieces of media content 
for which relevance feedback is received") & FIG. 2] and 

a combining process that creates a visual feature vector for each of the visual modalities and a 
textual feature vector for each of the textual modalities, and concatenates the visual feature 
vectors and the textual feature vectors into a unified feature vector. [0019] In the illustrated 
example, a client 102 can search media content store 104 for pieces of media content 106 that 
match a set of search criteria. This search criteria includes both low-level features and high- 
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level features. Low-level features are features that describe various low-level characteristics of 
the media content piece. For example, low-level features for image content may include color, 
texture, and shape features. High-level features are text features that are extracted from text 
associated with the media content piece, as discussed in more detail below. These low-level and 
high-level features corresponding to the media content piece 106 are compared to the set of 
search criteria to determine how closely the respective features match the set of search criteria. 
The results of these comparisons are then combined and the value resulting from the 
combination is used to determine how well the media content matches the set of search criteria. 

Regarding claim 2: 

Li et al teaches, 

A system, as in claim 1, further comprising a classifier induction process that induces a classifier 
from the unified feature vectors. [0027] In one implementation, classifier 132 uses a set of rules 
to classify each of the pieces of media content as either meaningful or not meaningful. These 
rules are based on various information regarding the media content, such as a color histogram 
of an image (e.g., an image that is predominately one color is not meaningful), the size of an 
image (e.g., an image less than a threshold size (such as 32.times.32 pixels) is not meaningful), 
the type of file (e.g., an image file that is a banner is not meaningful), etc. The rules to be used 
can vary, and in one implementation are determined empirically. 
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Regarding claim 3: 

Li et al teaches, 

A system, as in claim 2, where the classifiers include any one or more of the following: a 
hyperplane classifier, a rule-based classifier, a Bayesian classifier, maximum likelihood 
classifier. [0027] In one implementation, classifier 132 uses a set of rules to classify each of the 
pieces of media content as either meaningful or not meaningful These rules are based on 
various information regarding the media content, such as a color histogram of an image (e.g., an 
image that is predominately one color is not meaningful), the size of an image (e.g., an image 
less than a threshold size (such as 32 .times. 32 pixels) is not meaningful), the type of file (e.g., an 
image file that is a banner is not meaningful), etc. The rules to be used can vary, and in one 
implementation are determined empirically. 

Regarding claim 14: 

Li et al. teaches, 

A system, as in claim 1, where the visual feature vectors represent any one or more of the 
following: 

a color, a motion, a visual texture, an. optical flow, a semantic meaning, semantic meanings 
derived from one or more video streams, an edge density, a hue, an amplitude, a frequency, 
and a brightness. [0019] In the illustrated example, a client 102 can search media content store 
104 for pieces of media content 106 that match a set of search criteria. This search criteria 
includes both low-level features and high-level features. Low-level features are features that 
describe various low-level characteristics of the media content piece. For example, low-level 
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features for image content may include color, texture, and shape features. High-level features 
are text features that are extracted from text associated with the media content piece, as 
discussed in more detail below. These low-level and high-level features corresponding to the 
media content piece 106 are compared to the set of search criteria to determine how closely the 
respective features match the set of search criteria. The results of these comparisons are then 
combined and the value resulting from the combination is used to determine how well the media 
content matches the set of search criteria. 

Regarding claim 15: 

Li et al teaches, 

A system, as in claim 1, where the textual feature vectors are derived from any one or more of 
the following: 

close captions, open captions, captions, speech recognition applied to one or more audio input, 
semantic meanings derived from one or more audio streams, and global text information 
associated with the media item. [0035] (2) Image annotation: each image can have a 
corresponding image annotation which is a text label describing the semantics of the image, 
typically input by the creator of the image file. As this image annotation is intended to describe 
the semantics of the image, it typically includes valuable information describing the image. Thus, 
each word in the image annotation is a key feature (although certain common words and/or 
insignificant characters/character groups may be excluded as discussed above regarding image 
filenames and identifiers). 
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Regarding claim 22: 

Li et al teaches, 

A method for segmenting multimedia streams comprising the steps o£: 
storing one or more multimedia items in one or more memories of computer, each multimedia 
item having two or more disparate modalities, the disparate modalities being at least one or more 
visual modalities and one or more textual modalities; [Abstract] "Text features corresponding to 
pieces of media content (e.g., images, audio, multimedia content, etc.) are extracted from media 
content sources. One or more text features (e.g., one or more words) for a piece of media content 
are extracted from text associated with the piece of media content and text feature vectors 
generated therefrom and used during subsequent searching. Additional low-level feature vectors 
may also be extracted from the piece of media content and used during the subsequent searching. 
Relevance feedback can also be received from a user(s) identifying the relevance of pieces of 
media content rendered to the user in response to his or her search request The relevance 
feedback is logged and can be used in determining how to respond to subsequent search 
requests, such as by modifying feature vectors (e.g., text feature vectors) corresponding to the 
pieces of media content for which relevance feedback is received.")] 

dividing the multimedia items into blocks of one or more key intervals, each key interval having 
one more frames of the multimedia items; [0018] Media content store 104 represents a set of 
one or more sources from which media content can be received by a client 102. The storage of 
media content pieces 106 within media content store 104 can be arranged in any of a wide 
variety of manners and according to any of a wide variety of formats. For example, media 
content pieces 106 may be stored on multiple servers accessible using HTTP (Hypertext Transfer 
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Protocol). Media content pieces 106 can be any of a wide variety of conventional media content, 
such as audio content, video content (for example, still images or frames of motion video), 
multimedia content, etc. A piece of media content refers to media content that can be rendered, 
such as a single visual image, an audio clip (e.g., a song or portion of a song), a multimedia clip 
(e.g, an audio/video program or portion of an audio/video program), etc. Although discussed 
primarily with reference to images, the invention can be used with a wide variety of conventional 
media content. 

for each block, creating a visual feature vector for each of the visual modalities and a textual 
feature vector for each of the textual modalities; [Abstract] "Text features corresponding to 
pieces of media content (e.g., images, audio, multimedia content, etc.) are extracted from media 
content sources. One or more text features (e.g, one or more words) for a piece of media content 
are extracted from text associated with the piece of media content and text feature vectors 
generated therefrom and used during subsequent searching. Additional low-level feature vectors 
may also be extracted from the piece of media content and used during the subsequent searching. 
Relevance feedback can also be received from a user(s) identifying the relevance of pieces of 
media content rendered to the user in response to his or her search request. The relevance 
feedback is logged and can be used in determining how to respond to subsequent search 
requests, such as by modifying feature vectors (e.g, text feature vectors) corresponding to the 
pieces of media content for which relevance feedback is received.) & FIG. 2] 
for each block, concatenating the visual feature vectors and the textual feature vectors into a 
unified feature vector; [0041] Media content indexer 136 takes the extracted features for an 
image from media content and features database 140 and indexes the media content piece. The 
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indexing process refers to generating, as necessary, feature vectors corresponding to the media 
content piece and storing a correlation between the generated feature vectors and the media 
content piece. These generated feature vectors can be stored in database 140 or alternatively 
elsewhere. For low-level features, the extracted features are each a feature vector that is stored 
in database 140 by feature extractor 134, and thus no additional extraction or generation by 
indexer 136 is necessary. Alternatively, indexer 136 may combine (e.g., concatenate) the 
individual elements of each low-level feature vector for an image into a single low-level feature 
vector for the image. 

categorizing each of the blocks by categorizing the respective unified feature vector; [0047] 
Returning to act 156, if the media content piece is classified as meaningful, then low-level 
features are extracted from the media content piece and low-level feature vectors generated (act 
162). Additionally, high-level features are extracted from the media content piece and high-level 
feature vectors generated (act 164). These extracted feature vectors (both low-level and high- 
level) are then made available for searching (act 166). A check is then made as to whether there 
are additional media content pieces available from the source (act 160), and processing 
continues at either act 150 or act 154 accordingly, and 

assembling two or more of the categorized blocks into a segment. [0064] Initially, search criteria 
are received (act 200). The search criteria are converted to high-level and/or low-level query 
vectors (act 202). Assuming both high-level and low-level query vectors are created, the low- 
level query vector is compared to the low-level feature vectors of the media content pieces (act 
204), and the high-level query vector is compared to the high-level feature vectors of the media 
content pieces (act 206). The results of the comparisons in acts 204 and 206 are then combined 
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(act 208) and the media content pieces with the highest probability of being relevant (those most 
similar to the search criteria) are identified (act 210). The identified media content pieces are 
then rendered (act 212). 

Regarding claim 23: 

Li et al teaches, 

A memory storing a program, the program comprising the steps of: 

storing one or more multimedia items in one or more memories of computer, each multimedia 
item having two or more disparate modalities, the disparate modalities being at least one or more 
visual modalities and one or more textual modalities; [Abstract] "Text features corresponding to 
pieces of media content (e.g., images, audio, multimedia content, etc.) are extracted from media 
content sources. One or more text features (e.g., one or more words) for a piece of media content 
are extracted from text associated with the piece of media content and text feature vectors 
generated therefrom and used during subsequent searching. Additional low-level feature vectors 
may also be extracted from the piece of media content and used during the subsequent searching. 
Relevance feedback can also be received from a user(s) identifying the relevance of pieces of 
media content rendered to the user in response to his or her search request. The relevance 
feedback is logged and can be used in determining how to respond to subsequent search 
requests, such as by modifying feature vectors (e.g., text feature vectors) corresponding to the 
pieces of media content for which relevance feedback is received") & FIG. 2] 
dividing the multimedia items into block, of one or more key intervals, each key interval having 
one more frames of the multimedia items; [0018] Media content store 104 represents a set of 
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one or more sources from which media content can be received by a client 102. The storage of 
media content pieces 106 within media content store 104 can be arranged in any of a wide 
variety of manners and according to any of a wide variety of formats. For example, media 
content pieces 106 may be stored on multiple servers accessible using HTTP (Hypertext Transfer 
Protocol). Media content pieces 106 can be any of a wide variety of conventional media content, 
such as audio content, video content (for example, still images or frames of motion video), 
multimedia content, etc. A piece of media content refers to media content that can be rendered, 
such as a single visual image, an audio clip (e.g., a song or portion of a song), a multimedia clip 
(e.g., an audio/video program or portion of an audio/video program), etc. Although discussed 
primarily with reference to images, the invention can be used with a wide variety of conventional 
media content. 

for each block, creating a visual feature vector for each of the visual modalities and a textual 
feature vector for each of the textual modalities; [Abstract] "Text features corresponding to 
pieces of media content (e.g., images, audio, multimedia content, etc.) are extracted from media 
content sources. One or more text features (e.g., one or more words) for a piece of media content 
are extracted from text associated with the piece of media content and text feature vectors 
generated therefrom and used during subsequent searching. Additional low-level feature vectors 
may also be extracted from the piece of media content and used during the subsequent searching. 
Relevance feedback can also be received from a user(s) identifying the relevance of pieces of 
media content rendered to the user in response to his or her search request. The relevance 
feedback is logged and can be used in determining how to respond to subsequent search 
requests, such as by modifying feature vectors (e.g., text feature vectors) corresponding to the 
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pieces of media content for which relevance feedback is received,) & FIG. 2] 
for each block, concatenating the visual feature vectors and the textual feature vectors into a 
unified feature vector; [0041] Media content indexer 136 takes the extracted features for an 
image from media content and features database 140 and indexes the media content piece. The 
indexing process refers to generating, as necessary, feature vectors corresponding to the media 
content piece and storing a correlation between the generated feature vectors and the media 
content piece. These generated feature vectors can be stored in database 140 or alternatively 
elsewhere. For low -lev el features, the extracted features are each a feature vector that is stored 
in database 140 by feature extractor 134, and thus no additional extraction or generation by 
indexer 136 is necessary. Alternatively, indexer 136 may combine (e.g., concatenate) the 
individual elements of each low-level feature vector for an image into a single low-level feature 
vector for the image. 

categorizing each of the blocks by categorizing the respective unified feature vector; [0047] 
Returning to act 156, if the media content piece is classified as meaningful, then low-level 
features are extracted from the media content piece and low-level feature vectors generated (act 
162). Additionally, high-level features are extracted from the media content piece and high-level 
feature vectors generated (act 164). These extracted feature vectors (both low-level and high- 
level) are then made available for searching (act 166). A check is then made as to whether there 
are additional media content pieces available from the source (act 160), and processing 
continues at either act 150 or act 154 accordingly, and 

assembling two or more of the categorized blocks into a segment. [0064] Initially, search criteria 
are received (act 200). The search criteria are converted to high-level and/or low-level query 
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vectors (act 202), Assuming both high-level and low-level query vectors are created, the low- 
level query vector is compared to the low-level feature vectors of the media content pieces (act 
204), and the high-level query vector is compared to the high-level feature vectors of the media 
content pieces (act 206). The results of the comparisons in acts 204 and 206 are then combined 
(act 208) and the media content pieces with the highest probability of being relevant (those most 
similar to the search criteria) are identified (act 210). The identified media content pieces are 
then rendered (act 212). 

Regarding claim 24: 

Li et al teaches, 

A system for segmenting multimedia streams comprising: 

means for storing one or more multimedia items in one or more memories of computer, each 
multimedia item having two or more disparate modalities, the disparate modalities being at least 
one or more visual modalities and one or more textual modalities; [Abstract] ''Text features 
corresponding to pieces of media content (e.g } images, audio, multimedia content, etc.) are 
extracted from media content sources. One or more text features (e.g., one or more words) for a 
piece of media content are extracted from text associated with the piece of media content and 
text feature vectors generated therefrom and used during subsequent searching. Additional low- 
level feature vectors may also be extracted from the piece of media content and used during the 
subsequent searching. Relevance feedback can also be received from a user(s) identifying the 
relevance of pieces of media content rendered to the user in response to his or her search 
request The relevance feedback is logged and can be used in determining how to respond to 
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subsequent search requests, such as by modifying feature vectors (e.g., text feature vectors) 
corresponding to the pieces of media content for which relevance feedback is received") 
& FIG. 2] 

means for dividing the multimedia items into blocks of one or more key intervals, each key 
interval having one more frames of the multimedia items; [0018] Media content store 104 
represents a set of one or more sources from which media content can be received by a client 
102. The storage of media content pieces 106 within media content store 104 can be arranged in 
any of a wide variety of manners and according to any of a wide variety of formats. For example, 
media content pieces 106 may be stored on multiple servers accessible using HTTP (Hypertext 
Transfer Protocol). Media content pieces 106 can be any of a wide variety of conventional media 
content, such as audio content, video content (for example, still images or frames of motion 
video), multimedia content, etc. A piece of media content refers to media content that can be 
rendered, such as a single visual image, an audio clip (e.g., a song or portion of a song), a 
multimedia clip (e.g., an audio/video program or portion of an audio/video program), etc. 
Although discussed primarily with reference to images, the invention can be used with a wide 
variety of conventional media content. 

means for creating a visual feature vector for each of the visual modalities and a textual feature 
vector for each of the textual modalities, block by block; [0041] Media content indexer 136 takes 
the extracted features for an image from media content and features database 140 and indexes 
the media content piece. The indexing process refers to generating, as necessary, feature vectors 
corresponding to the media content piece and storing a correlation between the generated 
feature vectors and the media content piece. These generated feature vectors can be stored in 
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database 140 or alternatively elsewhere. For low-level features, the extracted features are each 
a feature vector that is stored in database 140 by feature extractor 134, and thus no additional 
extraction or generation by indexer 136 is necessary. Alternatively, indexer 136 may combine 
(e.g., concatenate) the individual elements of each low-level feature vector for an image into a 
single low-level feature vector for the image, 

means for concatenating the visual feature vectors and the textual feature vectors into a unified 
feature vector, block by block; [0041] Media content indexer 136 takes the extracted features for 
an image from media content and features database 140 and indexes the media content piece. 
The indexing process refers to generating, as necessary, feature vectors corresponding to the 
media content piece and storing a correlation between the generated feature vectors and the 
media content piece. These generated feature vectors can be stored in database 140 or 
alternatively elsewhere. For low-level features, the extracted features are each a feature vector 
that is stored in database 140 by feature extractor 134, and thus no additional extraction or 
generation by indexer 136 is necessary. Alternatively, indexer 136 may combine (e.g., 
concatenate) the individual elements of each low-level feature vector for an image into a single 
low-level feature vector for the image. 

means for categorizing each of the blocks by categorizing the respective unified feature vector; 
[0047] Returning to act 156, if the media content piece is classified as meaningful, then low-level 
features are extracted from the media content piece and low-level feature vectors generated (act 
162). Additionally, high-level features are extracted from the media content piece and high-level 
feature vectors generated (act 164). These extracted feature vectors (both low-level and high- 
level) are then made available for searching (act 166). A check is then made as to whether there 
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are additional media content pieces available from the source (act 160), and processing 

continues at either act 150 or act 154 accordingly. 

and 

means for assembling two or more of the categorized blocks into a segment. [0064] Initially, 
search criteria are received (act 200). The search criteria are converted to high-level and/or 
low-level query vectors (act 202). Assuming both high-level and low-level query vectors are 
created, the low-level query vector is compared to the low-level feature vectors of the media 
content pieces (act 204), and the high-level query vector is compared to the high-level feature 
vectors of the media content pieces (act 206). The results of the comparisons in acts 204 and 206 
are then combined (act 208) and the media content pieces with the highest probability of being 
relevant (those most similar to the search criteria) are identified (act 210). The identified media 
content pieces are then rendered (act 212). 



12. The prior art made of record and (listed of form PTO-892) not relied upon is considered 
pertinent to applicant's disclosure as follows. Applicant or applicant's representative is respect- 
fully reminded that in process of patent prosecution i.e., amending of claims in response to a 
rejection of claims set forth by the Examiner per Title 35 U.S.C. The patentable novelty must be 



Allowable Subject Matter 



Claims 16-21 are allowed. 



Conclusion 
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clearly shown in view of the state of the art disclosed by the references cited and any objections 
made. Moreover, applicant or applicant's representative must clearly show how the amendments 
avoid or overcome such references and objections. See 37 CFR § 1.111(c). 



Correspondence Information 

13. Any inquiries concerning this communication or earlier communications from the 
examiner should be directed to Michael B. Holmes who may be reached via telephone at 
(703) 308-6280. The examiner can normally be reached Monday through Friday between 
8:00 a.m. and 5:00 p.m. eastern standard time. 

If you need to send the Examiner, a facsimile transmission regarding After Final 
issues, please send it to (703) 746-7238. If you need to send an Official facsimile trans- 
mission, please send it to (703) 746-7239. If you would like to send a Non-Official (draft) 
facsimile transmission the fax is (703) 746-7240. If attempts to reach the examiner by tele- 
phone are unsuccessful, the Examiner's Supervisor, Anthony Knight, may be reached at 
(703) 308-3179. 

Any response to this office action should be mailed too: 

Director of Patents and Trademarks Washington, D.C. 20231. Hand-delivered 
responses should be delivered to the Receptionist, located on the fourth floor of 
Crystal Park II, 2121 Crystal Drive Arlington, Virginia. 
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